feat: add Google Cloud Storage support (GCS) #2193

shadinaif · 2024-03-17T07:10:03Z

TL;DR - This is to add support to GCS files backend by adding a new option (gcs) to ORA2_FILEUPLOAD_BACKEND

JIRA: ---

What changed?

ORA2_FILEUPLOAD_BACKEND environment variable accepts a new value gcs in addition to the existing options (s3, filesystem, swift, and django)
No changes to the requirements. It is assumed that you already installed django-storages[google] instead of the default django-storages which will include google-cloud-storages

Important to know!
Using GSC with ORA means that edx-platform is already configured to use GCS instead of S3

It is assumed that you already installed django-storages[google] instead of the default django-storages which will include google-cloud-storages
It is assumed that DEFAULT_FILE_STORAGE is set to storages.backends.gcloud.GoogleCloudStorage in edx-platform. Other configurations might be needed too
It is assumed that edx-platform functions directly and seamlessly with GCS without a proxy like MinIO or boto3

Developer Checklist

Reviewed the release process
Translations and JS/SASS compiled~~ not needed
Bumped version number in openassessment/__init__.py and package.json

Testing Instructions

Using tutor development:

Prepare your GSC storage, make sure to have a dedicated bucket for file upload
Configure edx-platform to use GCS, including the installation of django-storages[google]
Configure ORA2 to use GCS by setting ORA2_FILEUPLOAD_BACKEND to gcs
Configure the dedicated bucket by setting FILE_UPLOAD_STORAGE_BUCKET_NAME
Test uploading files from ORA2 and previewing them
Test previewing uploaded files in ora2 MFE
Verify that files exist in the dedicated storage
Test deleting files
Verify that files were deleted from the dedicated storage

Reviewer Checklist

Collectively, these should be completed by reviewers of this PR:

I've done a visual code review
I've tested the new functionality

FYI: @openedx/content-aurora

openedx-webhooks · 2024-03-17T07:10:08Z

Thanks for the pull request, @shadinaif!

What's next?

Please work through the following steps to get your changes ready for engineering review:

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

🔘 Let us know that your PR is ready for review:

Who will review my changes?

This repository is currently maintained by @openedx/2u-aurora. Tag them in a comment and let them know that your changes are ready for review.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

itsjeyd · 2024-03-22T09:30:07Z

@shadinaif Thank you for this contribution! Let us know when the changes are ready for review.

itsjeyd · 2024-04-22T07:52:59Z

Hey @shadinaif, just checking in to see if you're still planning to continue working on this PR?

shadinaif · 2024-04-22T09:07:07Z

Thank you for the reminder @itsjeyd . Yes, it's on production and working fine. I'll rebase and set to review soon

itsjeyd · 2024-04-26T07:25:35Z

@shadinaif Sounds good, thanks for the update.

itsjeyd · 2024-05-23T15:01:26Z

Hey @shadinaif, do you have any updates about when you'll be able to come back to this PR?

shadinaif · 2024-05-25T13:06:39Z

Thanks for the follow-up @itsjeyd , I'll get back to it and complete it on Tuesday, June 4, 2024 I promise. I've added it to my schedule. Sorry for the delay 🙏🏼

itsjeyd · 2024-05-31T07:21:21Z

Sounds good, thanks for the update @shadinaif.

itsjeyd · 2024-06-06T07:57:42Z

@shadinaif It looks like tests will need some further tweaking here. Could you have a look?

codecov · 2024-06-07T13:34:00Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.12%. Comparing base (8b320d6) to head (8fd1c39).
Report is 31 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2193      +/-   ##
==========================================
+ Coverage   95.09%   95.12%   +0.02%     
==========================================
  Files         195      196       +1     
  Lines       21479    21584     +105     
  Branches     1931     1935       +4     
==========================================
+ Hits        20425    20531     +106     
  Misses        787      787              
+ Partials      267      266       -1

Flag	Coverage Δ
unittests	`95.12% <100.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

itsjeyd · 2024-06-26T11:39:40Z

Hey @shadinaif, thanks for working on the tests.

It looks like some of them are still failing. Will you be taking another look?

shadinaif · 2024-06-26T20:54:07Z

Thanks for the reminder @itsjeyd , I'll do in a couple of days 👍🏼

itsjeyd · 2024-07-18T15:06:45Z

@shadinaif Just checking in to see if you're still planning to continue working on this PR? If you don't think you'll get to it in the short term, it might make sense to close it for the time being.

shadinaif · 2024-07-26T15:14:49Z

tox.ini

@@ -7,7 +7,7 @@ deps =
    django42: Django>=4.2,<5.0

 commands =
-    pytest
+    pytest {posargs}


this is not related to the PR, it just makes it easier to run tests for a single file or object locally

[E0606(possibly-used-before-assignment), TeamMixin.add_team_submission_context] Possibly using variable team_submission before assignment [E0606(possibly-used-before-assignment), serialize_training_examples] Possibly using variable parts before assignment [E0606(possibly-used-before-assignment), StaffWorkflowListViewUnitTests.test_bulk_fetch_annotated_staff_workflows] Possibly using variable assessment_ids before assignment

shadinaif · 2024-07-27T15:39:42Z

@itsjeyd thank you for your patience on this. I've added the needed tests, and I also had to fix some linter errors not related to the PR. All green now

itsjeyd · 2024-08-08T06:54:34Z

@shadinaif Great! I'm marking this PR as ready for review now.

itsjeyd · 2024-08-08T06:58:11Z

Hi @openedx/2u-aurora team, this PR is ready for engineering review.

@feanil I can't formally request a review from the Aurora team, could you please update the team's permissions for this repo to make that possible?

feanil · 2024-08-12T14:01:17Z

Added the team to the repo and tagged them for review.

pomegranited

Hi @shadinaif , I'm having trouble getting Google Cloud credentials configured to test this change. Are you planning to submit a GCS integration PR to the platform?

I found documentation for initializing the GS_CREDENTIALS setting, but since it requires initializing that setting as a google.oauth2.service_account.Credentials object, I can't configure it with yml, and would have to modify the platform's envs/*.py files.

And without platform integration, I don't think we should add it to ORA2 either. Open to discussion though, let me know?

pomegranited · 2024-09-10T01:40:41Z

openassessment/xblock/team_mixin.py

@@ -134,7 +134,7 @@ def add_team_submission_context(
            raise TypeError("One of team_submission_uuid or individual_submission_uuid must be provided")
        if team_submission_uuid:
            team_submission = get_team_submission(team_submission_uuid)
-        elif individual_submission_uuid:
+        else:


This change looks unrelated?

pomegranited · 2024-09-10T01:41:06Z

openassessment/xblock/utils/xml.py

@@ -625,6 +625,7 @@ def serialize_training_examples(examples, assessment_el):
        answer_el = etree.SubElement(example_el, 'answer')
        try:
            answer = example_dict.get('answer')
+            parts = []


This change looks unrelated?

pomegranited · 2024-09-10T01:43:07Z

requirements/quality.txt

@@ -2,7 +2,7 @@
 # This file is autogenerated by pip-compile with Python 3.8
 # by the following command:
 #
-#    make upgrade
+#    pip-compile --output-file=requirements/quality.txt requirements/quality.in


nit: This repo is missing the compile-requirements make target, but would you be willing to add it with the diff below? Then you can run make compile-requirements and avoid this unneeded change to the requirements file.

diff --git a/Makefile b/Makefile index 68cc2a636..aa2ca0bcf 100644 --- a/Makefile +++ b/Makefile @@ -43,23 +43,23 @@ $(COMMON_CONSTRAINTS_TXT): wget -O "$(@)" https://raw.githubusercontent.com/edx/edx-lint/master/edx_lint/files/common_constraints.txt || touch "$(@)" echo "$(COMMON_CONSTRAINTS_TEMP_COMMENT)" | cat - $(@) > temp && mv temp $(@) -upgrade: export CUSTOM_COMPILE_COMMAND=make upgrade -upgrade: $(COMMON_CONSTRAINTS_TXT) ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in +compile-requirements: export CUSTOM_COMPILE_COMMAND=make upgrade +compile-requirements: $(COMMON_CONSTRAINTS_TXT) ## update the requirements/*.txt files with the latest packages satisfying requirements/*.in # global common_constraints has this pin. sed 's/django-simple-history==3.0.0//g' requirements/common_constraints.txt > requirements/common_constraints.tmp mv requirements/common_constraints.tmp requirements/common_constraints.txt pip install -qr requirements/pip-tools.txt - pip-compile --upgrade --allow-unsafe -o requirements/pip.txt requirements/pip.in - pip-compile --upgrade -o requirements/pip-tools.txt requirements/pip-tools.in + pip-compile ${COMPILE_OPTS} --allow-unsafe -o requirements/pip.txt requirements/pip.in + pip-compile ${COMPILE_OPTS} -o requirements/pip-tools.txt requirements/pip-tools.in pip install -qr requirements/pip.txt pip install -qr requirements/pip-tools.txt - pip-compile --upgrade -o requirements/base.txt requirements/base.in - pip-compile --upgrade -o requirements/test.txt requirements/test.in - pip-compile --upgrade -o requirements/quality.txt requirements/quality.in - pip-compile --upgrade -o requirements/test-acceptance.txt requirements/test-acceptance.in - pip-compile --upgrade -o requirements/tox.txt requirements/tox.in - pip-compile --upgrade -o requirements/ci.txt requirements/ci.in - pip-compile --upgrade -o requirements/docs.txt requirements/docs.in + pip-compile ${COMPILE_OPTS} -o requirements/base.txt requirements/base.in + pip-compile ${COMPILE_OPTS} -o requirements/test.txt requirements/test.in + pip-compile ${COMPILE_OPTS} -o requirements/quality.txt requirements/quality.in + pip-compile ${COMPILE_OPTS} -o requirements/test-acceptance.txt requirements/test-acceptance.in + pip-compile ${COMPILE_OPTS} -o requirements/tox.txt requirements/tox.in + pip-compile ${COMPILE_OPTS} -o requirements/ci.txt requirements/ci.in + pip-compile ${COMPILE_OPTS} -o requirements/docs.txt requirements/docs.in # Delete django pin from test requirements to avoid tox version collision sed -i.tmp '/^[d|D]jango==/d' requirements/test.txt sed -i.tmp '/^djangorestframework==/d' requirements/test.txt @@ -70,6 +70,9 @@ upgrade: $(COMMON_CONSTRAINTS_TXT) ## update the requirements/*.txt files with # Delete temporary files rm requirements/*.txt.tmp +upgrade: ## update the pip requirements files to use the latest releases satisfying our constraints + $(MAKE) compile-requirements COMPILE_OPTS="--upgrade" +

itsjeyd · 2024-09-20T12:26:26Z

Thanks for picking up the review for this PR @pomegranited!

itsjeyd · 2024-10-17T07:38:43Z

Hey @shadinaif, just checking in to see if you're still planning to continue working on this PR?

shadinaif · 2024-10-17T07:53:27Z

Hi @itsjeyd , looks like I missed the notification for the review. I'll continue with it on the weekend. Thank you for the ping!

itsjeyd · 2024-11-14T10:40:21Z

Hi @shadinaif, it's been about a month without any activity here, so I'm closing this PR for now. Feel free to reopen it (or open a new one) when you're ready to come back to this work.

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Mar 17, 2024

shadinaif force-pushed the shadinaif/add-gcs-support branch from 5fc79b6 to 81c511c Compare April 23, 2024 04:14

shadinaif force-pushed the shadinaif/add-gcs-support branch 2 times, most recently from 7a5ecbb to c520148 Compare June 4, 2024 16:15

shadinaif marked this pull request as ready for review June 4, 2024 16:21

shadinaif requested a review from a team as a code owner June 4, 2024 16:21

shadinaif force-pushed the shadinaif/add-gcs-support branch 2 times, most recently from 089d5f4 to e41bab5 Compare June 7, 2024 13:28

itsjeyd removed the request for review from a team June 26, 2024 11:36

shadinaif force-pushed the shadinaif/add-gcs-support branch 2 times, most recently from 76a4a62 to 0461792 Compare July 26, 2024 15:13

shadinaif commented Jul 26, 2024

View reviewed changes

shadinaif force-pushed the shadinaif/add-gcs-support branch from 0461792 to 1e9c989 Compare July 26, 2024 15:25

shadinaif added 2 commits July 26, 2024 19:11

feat: add Google Cloud Storage support (GCS)

9cf4b5b

shadinaif force-pushed the shadinaif/add-gcs-support branch from eb297fd to 8fd1c39 Compare July 26, 2024 16:11

itsjeyd added the waiting for eng review PR is ready for review. Review and merge it, or suggest changes. label Aug 8, 2024

feanil requested a review from a team August 12, 2024 14:01

itsjeyd added the needs reviewer assigned PR needs to be (re-)assigned a new reviewer label Sep 7, 2024

pomegranited self-assigned this Sep 10, 2024

pomegranited requested changes Sep 10, 2024

View reviewed changes

itsjeyd added waiting on author PR author needs to resolve review requests, answer questions, fix tests, etc. and removed needs reviewer assigned PR needs to be (re-)assigned a new reviewer waiting for eng review PR is ready for review. Review and merge it, or suggest changes. labels Sep 20, 2024

itsjeyd added the inactive PR author has been unresponsive for several months label Oct 17, 2024

itsjeyd removed the inactive PR author has been unresponsive for several months label Oct 17, 2024

itsjeyd closed this Nov 14, 2024

itsjeyd added closed inactivity PR was closed because the author abandoned it and removed waiting on author PR author needs to resolve review requests, answer questions, fix tests, etc. labels Nov 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Google Cloud Storage support (GCS) #2193

feat: add Google Cloud Storage support (GCS) #2193

shadinaif commented Mar 17, 2024 •

edited

Loading

openedx-webhooks commented Mar 17, 2024 •

edited

Loading

itsjeyd commented Mar 22, 2024

itsjeyd commented Apr 22, 2024

shadinaif commented Apr 22, 2024

itsjeyd commented Apr 26, 2024

itsjeyd commented May 23, 2024

shadinaif commented May 25, 2024

itsjeyd commented May 31, 2024

itsjeyd commented Jun 6, 2024

codecov bot commented Jun 7, 2024 •

edited

Loading

itsjeyd commented Jun 26, 2024

shadinaif commented Jun 26, 2024

itsjeyd commented Jul 18, 2024

shadinaif Jul 26, 2024 •

edited

Loading

shadinaif commented Jul 27, 2024

itsjeyd commented Aug 8, 2024

itsjeyd commented Aug 8, 2024

feanil commented Aug 12, 2024

pomegranited left a comment

pomegranited Sep 10, 2024

pomegranited Sep 10, 2024

pomegranited Sep 10, 2024 •

edited

Loading

itsjeyd commented Sep 20, 2024

itsjeyd commented Oct 17, 2024

shadinaif commented Oct 17, 2024

itsjeyd commented Nov 14, 2024

feat: add Google Cloud Storage support (GCS) #2193

feat: add Google Cloud Storage support (GCS) #2193

Conversation

shadinaif commented Mar 17, 2024 • edited Loading

openedx-webhooks commented Mar 17, 2024 • edited Loading

What's next?

🔘 Get product approval

🔘 Provide context

🔘 Get a green build

🔘 Let us know that your PR is ready for review:

Who will review my changes?

Where can I find more information?

When can I expect my changes to be merged?

itsjeyd commented Mar 22, 2024

itsjeyd commented Apr 22, 2024

shadinaif commented Apr 22, 2024

itsjeyd commented Apr 26, 2024

itsjeyd commented May 23, 2024

shadinaif commented May 25, 2024

itsjeyd commented May 31, 2024

itsjeyd commented Jun 6, 2024

codecov bot commented Jun 7, 2024 • edited Loading

Codecov Report

itsjeyd commented Jun 26, 2024

shadinaif commented Jun 26, 2024

itsjeyd commented Jul 18, 2024

shadinaif Jul 26, 2024 • edited Loading

Choose a reason for hiding this comment

shadinaif commented Jul 27, 2024

itsjeyd commented Aug 8, 2024

itsjeyd commented Aug 8, 2024

feanil commented Aug 12, 2024

pomegranited left a comment

Choose a reason for hiding this comment

pomegranited Sep 10, 2024

Choose a reason for hiding this comment

pomegranited Sep 10, 2024

Choose a reason for hiding this comment

pomegranited Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

itsjeyd commented Sep 20, 2024

itsjeyd commented Oct 17, 2024

shadinaif commented Oct 17, 2024

itsjeyd commented Nov 14, 2024

shadinaif commented Mar 17, 2024 •

edited

Loading

openedx-webhooks commented Mar 17, 2024 •

edited

Loading

codecov bot commented Jun 7, 2024 •

edited

Loading

shadinaif Jul 26, 2024 •

edited

Loading

pomegranited Sep 10, 2024 •

edited

Loading